IBI  ■  Defence  Research  and  Recherche  et  developpement 
B^B  Development  Canada  pour  la  defense  Canada 


A  training  study  of  the  Hercules 
Observer  Trainer 


Stuart  C.  Grant 


Defence  R&D  Canada 

Technical  Report 
DRDC  Toronto  TR  2009-008 
April  2009 


Canada 


A  training  study  of  the  Hercules  Observer 
Trainer 


Stuart  C. Grant 


Defence  R&D  Canada  -  Toronto 

Technical  Report 

DRDC  Toronto  TR  2009-008 

April  2009 


Principal  Author 

Original  signed  by  Stuart  C.  Grant 


Stuart  C.  Grant 

Defence  Scientist,  Collaborative  Behaviour  and  Learning  Section 

Approved  by 

Original  signed  by  Keith  Stewart 
Keith  Stewart 

Head,  Collaborative  Behaviour  and  Learning  Section 

Approved  for  release  by 

Original  signed  by  K.C.  Wulterkens 
K.C.  Wulterkens 

for  Chair,  Document  Review  and  Library  Committee 


Sponsored  by  the  Canadian  Forces  Aerospace  Warfare  Centre. 


In  conducting  the  research  described  in  this  report,  the  investigators  adhered  to  the  policies  and 
procedures  set  out  in  the  Tri-Council  Policy  Statement:  Ethical  conduct  for  research  involving 
humans,  National  Council  on  Ethics  in  Human  Research,  Ottawa,  1998  as  issued  jointly  by  the 
Canadian  Institutes  of  Health  Research,  the  Natural  Sciences  and  Engineering  Research  Council 
of  Canada  and  the  Social  Sciences  and  Humanities  Research  Council  of  Canada. 


©  Her  Majesty  the  Queen  in  Right  of  Canada,  as  represented  by  the  Minister  of  National  Defence,  2009 


©  Sa  Majeste  la  Reine  (en  droit  du  Canada),  telle  que  representee  par  le  ministre  de  la  Defense  nationale, 
2009 


Abstract 


The  Canadian  Forces  (CF)  operates  aircraft  in  theatres  where  they  are  threatened  by  missiles  and 
gunfire.  Timely,  coherent  and  accurate  reporting  of  the  threats  is  vital  to  defeating  them.  A 
training  device  called  the  Hercules  Observer  Trainer  (HOT)  was  developed  to  support  the  training 
of  surface-to-air  threat  reporting.  To  validate  the  simulation  and  determine  its  training 
effectiveness,  an  experiment  employing  16  CF  air  crewmembers  was  conducted  at  CFB  Trenton. 
Half  of  the  experimental  subjects  were  qualified  tactical  aircrew  and  were  deemed  experts  at 
making  threat  calls.  The  other  half  consisted  of  students  on  a  CC-130  Basic  Loadmaster  course 
who  had  not  yet  qualified  as  loadmasters  on  the  CC-130  and  were  deemed  novices.  Both  groups 
of  subjects  completed  two  sessions  of  making  threat  calls  against  26  surface  to  air  missile  and  24 
anti-aircraft  artillery  threats.  In  the  first  session,  experts  made  more  accurate  and  more 
syntactically  correct  threat  calls  than  novices.  By  the  end  of  the  second  session,  both  groups 
showed  improved  accuracy  and  syntax  over  their  earlier  levels  of  performance,  and  the 
performance  of  the  novices  was  equivalent  to  that  of  the  experts.  It  is  concluded  that  the  HOT  is  a 
valid  simulation  of  the  threat  call  task,  that  students  can  use  the  HOT  to  learn  the  task,  and  that 
the  training  is  transferrable  to  the  operational  environment.  It  is  recommended  that  HOT  be 
considered  for  operational  training  in  the  CC-130  community.  It  is  further  recommended  that  the 
training  scenarios  be  adapted  and  the  technology  improved  to  best  represent  the  current 
operational  environment. 


Resume 


Les  Forces  canadiennes  (FC)  utilisent  des  aeronefs  qui,  dans  certains  theatres,  sont  menaces  par 
des  tirs  de  missiles  et  de  l’artillerie.  Des  rapports  rapides,  coherents  et  precis  sur  les  menaces  sont 
essentiels  pour  les  contrer.  Un  appareil  appele  observateur  d’entrainement  du  Hercules  (HOT)  a 
ete  mis  au  point  pour  donner  de  la  formation  sur  Fetablissement  de  rapports  et  les  fa9ons 
d’intervenir  en  cas  de  menaces  sol-air.  Afm  de  valider  l’appareil  et  d’etablir  l’efficacite  de  sa 
capacite  de  formation,  une  experience  a  ete  realisee  a  la  BFC  Trenton,  a  laquelle  seize  membres 
du  personnel  navigant  des  FC  ont  participe.  La  moitie  des  sujets  de  F  experience  faisaient  partie 
de  F equipage  aerien  tactique  et  etaient  consideres  comme  des  specialistes  de  la  production  de 
messages  signalant  une  menace.  L’autre  moitie  etait  composee  de  stagiaires  du  cours  elementaire 
d’arrimeur  du  CC-130.  Ils  n’avaient  pas  encore  la  qualification  d’arrimeur  et  etaient  consideres 
comme  des  novices.  Les  deux  groupes  de  sujets  ont  participe  a  deux  seances  de  production  de 
messages  signalant  la  menace  de  26  missiles  sol-air  et  de  24  tirs  d’artillerie  antiaerienne.  Lors  de 
la  premiere  seance,  les  specialistes  ont  signale  les  menaces  de  maniere  plus  precise  et  plus  exacte 
au  plan  syntaxique  que  les  novices.  A  la  fin  de  la  deuxieme  seance,  les  deux  groupes  s’ etaient 
ameliores  et  avait  un  rendement  equivalent  sur  les  deux  aspects.  On  peut  done  conclure  que 
l’appareil  HOT  permet  une  simulation  valide  de  la  production  de  messages  signalant  une  menace 
et  que  la  formation  peut  etre  transferee  dans  un  environnement  operationnel.  On  recommande 
done  l’appareil  HOT  pour  l’instruction  operationnelle  de  la  collectivite  du  CC-130.  On 
recommande  egalement  l’adaptation  des  scenarios  d’entrainement  et  F  amelioration  de  la 
technologie  afin  de  mieux  representer  le  contexte  operationnel  actuel. 
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Executive  summary 


A  training  study  of  the  Hercules  Observer  T rainer 

Stuart  C.  Grant;  DRDC  Toronto  TR  2009-008;  Defence  R&D  Canada  -  Toronto; 
April  2009. 

Introduction: 

The  Canadian  Forces  (CF)  operates  aircraft  in  theatres  where  they  are  threatened  by  missiles  and 
gunfire.  CC-130  transport  aircraft  operations  in  combat  areas  employ  visual  scanning  by  aircrew 
to  detect  and  coordinate  reaction  to  threats.  The  threats  include  small  arms,  anti-aircraft  artillery 
(AAA),  surface  to  air  missiles  (SAMs),  helicopters  (HELO),  and  fixed  wing  threats.  Observers  in 
the  aircraft  report  the  existence  of  the  threats,  provide  advice  on  the  required  reaction,  and  report 
the  location  of  the  threat.  Their  reports  must  be  fast,  accurate,  and  consistent  for  their  aircraft  to 
respond  in  a  timely  and  appropriate  manner. 

Currently,  the  CF  has  limited  means  to  train  aircrew  in  the  recognition  and  avoidance  of  these 
threats.  Computer  simulation  of  ground  threats  is  not  currently  available,  live  training  is  very 
primitive,  and  the  measures  of  effectiveness  are  limited  to  instructor  observations.  The 
commander  of  8  Wing  therefore  raised  a  Statement  of  Operational  Capability  Deficiency 
endorsed  by  1  Can  Air  Div,  to  obtain  training  devices  to  address  the  deficiency.  In  parallel,  the 
Canadian  Forces  Aerospace  Warfare  Centre  (CFAWC)  tasked  DRDC  Toronto  with  developing  a 
prototype  device  to  provide  CC-130  loadmasters  with  threat  reaction  training.  DRDC  performed  a 
task  analysis,  a  training  requirements  analysis,  and  specified  a  technical  solution  that  was 
implemented  under  a  series  of  contracts.  The  experiment  reported  in  this  document  provides  an 
evaluation  of  the  training  effectiveness  of  the  resulting  Hercules  Observer  Trainer  (HOT). 

To  validate  the  HOT  and  determine  its  training  effectiveness,  an  experiment  employing  16  CF  air 
crewmembers  was  conducted  at  CFB  Trenton.  Half  of  the  experimental  subjects  were 
qualified  tactical  aircrew  and  were  deemed  experts  at  making  threat  calls.  The  other  half 
consisted  of  students  on  a  CC-130  Basic  Loadmaster  course  who  had  not  yet  qualified  as 
loadmasters  on  the  CC-130  and  were  deemed  novices.  Both  groups  of  subjects  completed  two 
sessions,  each  of  which  involved  making  threat  calls  against  26  surface  to  air  missile  and  24  anti¬ 
aircraft  artillery  threats. 

Results: 

In  the  first  session,  experts  made  significantly  more  accurate  and  more  syntactically  correct  threat 
calls  than  novices.  Furthermore,  both  groups  of  subjects  improved  on  both  measures  at  the  end  of 
the  second  session.  At  the  conclusion  of  the  second  session,  performance  of  the  two  groups  was 
equivalent  on  both  measures.  It  is  concluded  that  the  HOT  is  a  valid  simulation  of  the  threat  call 
task,  that  students  can  use  the  HOT  to  learn  the  task,  and  that  the  training  is  transferable  to  the 
operational  environment. 


iii 


DRDC  Toronto  TR  2009-008 


Significance: 


The  experiment  revealed  that  the  HOT  is  a  valid  representation  of  the  threat  environment,  shown 
by  expert  aircrew  initially  out-performing  novices  in  the  threat  reaction  task.  The  HOT 
demonstrated  its  ability  to  train  and  to  lead  to  expert  performance  when  the  performance  of 
novices  improved  and  came  to  be  indistinguishable  from  that  of  the  experts.  The  HOT  provides 
the  CF  with  a  device  and  Crown-owned  intellectual  property  for  training  threat  reaction  where 
none  existed  before. 

Future  plans: 

Several  improvements  can  be  made  to  the  HOT  to  increase  the  utility  of  the  existing  version  of 
the  device.  The  visual  imagery  can  be  optimally  rendered  for  the  existing  projection  surface. 
Also,  the  time  for  a  student  to  respond  to  a  threat  could  be  made  available  to  the  instructor  in  a 
reliable  manner.  Finally,  the  appearance  of  the  tracer  fire  and  missile  trails  can  be  made  more 
realistic  and  variable. 

Future  generations  of  the  HOT  that  provide  night  viewing  conditions  and  night  vision  goggle 
simulation  (or  night  vision  goggle  stimulation)  must  be  subject  to  a  validation  study. 
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A  training  study  of  the  Hercules  Observer  T rainer 

Stuart  C.  Grant;  DRDC  Toronto  TR  2009-008;  R  &  D  pour  la  defense  Canada  - 
Toronto;  Avril  2009. 

Introduction  : 

Les  Forces  canadiennes  (FC)  utilisent  des  aeronefs  qui,  dans  certains  theatres,  sont  menaces  par 
des  tirs  de  missiles  et  de  l’artillerie.  Lorsque  les  avions  de  transport  CC-130  survolent  les  zones 
de  combat,  on  fait  appel  au  balayage  visuel  des  membres  du  personnel  navigant  pour  detecter  les 
menaces  et  coordonner  les  mesures  pouvant  les  contrer.  Precisons  notamment  la  menace  des 
armes  legeres,  de  l’artillerie  antiaerienne  (AAA),  des  missiles  sol-air  (SAM),  des  helicopteres 
(HELO)  et  des  aeronefs  a  voilure  fixe.  Les  observateurs  a  bord  des  avions  signalent  la  presence 
d’une  menace,  donnent  des  conseils  sur  les  mesures  a  prendre  et  precisent  d’ou  elle  provient. 
Leurs  rapports  doivent  etre  rapides,  precis  et  coherents  afm  que  l’avion  puisse  reagir  en  temps 
opportun  et  de  maniere  appropriee. 

A  l’heure  actuelle,  les  FC  disposent  de  moyens  limites  pour  entrainer  des  equipages  navigants  a 
reconnaitre  et  a  eviter  ces  menaces.  La  simulation  informatique  des  menaces  terrestres  n’est  pas 
encore  disponible,  l’entrainement  en  situation  reelle  n’est  qu’au  tout  debut  et  les  mecanismes 
servant  a  mesurer  l’efficacite  sont  reserves  aux  observations  de  l’instructeur.  Le  commandant  de 
la  8e  Escadre  a  done  formule  un  enonce  d'insuffisance  de  capacites  operationnelles,  appuye  par  la 
1  DAC,  afm  d’obtenir  des  appareils  d’entrainement  pour  pallier  les  lacunes.  Parallelement,  le 
Centre  de  guerre  aerospatiale  des  Forces  canadiennes  (CGAFC)  a  charge  RDDC  Toronto  de 
mettre  au  point  le  prototype  d’un  appareil  qui  permettrait  de  donner  aux  arrimeurs  du  CC-130  une 
formation  sur  la  fa9on  de  reagir  en  cas  de  menaces.  RDDC  a  precede  a  une  analyse  des  taches  et 
des  exigences  en  matiere  de  formation,  et  a  determine  une  solution  technique  qui  a  ete  mise  en 
oeuvre  dans  toute  une  gamme  de  contrats.  L’experience  dont  il  est  question  dans  le  present 
document  renferme  une  evaluation  de  l’efficacite  de  la  formation  que  permet  l’observateur 
d’entrainement  du  Hercules  (HOT). 

Afm  de  valider  l’appareil  et  d’etablir  l’efficacite  de  sa  capacite  de  formation,  une  experience  a  ete 
realisee  a  la  BFC  Trenton,  a  laquelle  seize  membres  du  personnel  navigant  des  FC  ont  participe. 
La  moitie  des  sujets  de  l’experience  faisaient  partie  de  l’equipage  aerien  tactique  et  etaient 
consideres  comme  des  specialistes  de  la  production  de  messages  signalant  une  menace.  L’autre 
moitie  etait  composee  de  stagiaires  du  cours  elementaire  d’arrimeur  du  CC-130.  Ils  n’avaient  pas 
encore  la  qualification  d’arrimeur  et  etaient  consideres  comme  des  novices.  Les  deux  groupes  de 
sujets  ont  participe  a  deux  seances  de  production  de  messages  signalant  la  menace  de  26  missiles 
sol-air  et  de  24  tirs  d’artillerie  antiaerienne. 

Resultats  : 

Lors  de  la  premiere  seance,  les  specialistes  ont  signale  les  menaces  de  maniere  beaucoup  plus 
precise  et  plus  exacte  au  plan  syntaxique  que  les  novices.  En  outre,  a  la  fin  de  la  deuxieme  seance, 
les  deux  groupes  s’etaient  ameliores  et  leur  rendement  etait  equivalent  sur  les  deux  aspects.  On 
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peut  done  cone  lure  que  l’appareil  HOT  permet  une  simulation  valide  de  la  production  de 
messages  signalant  une  menace,  que  les  stagiaires  peuvent  utiliser  Tappareil  pour  apprendre  ce 
qu’ils  ont  a  faire  et  que  la  formation  peut  etre  transferee  dans  un  environnement  operationnel. 

Importance  : 

L’experience  a  revele  que  Tappareil  HOT  permet  une  representation  valide  d’un  environnement 
de  menace,  illustrant  comment  des  specialistes  du  personnel  navigant  ont  d’abord  declasse  des 
novices  relativement  aux  actions  a  prendre  dans  une  situation  de  menace.  L’appareil  HOT  a 
montre  sa  capacite  de  formation  et  sa  capacite  a  obtenir  un  rendement  de  specialiste  en  ameliorant 
celui  des  novices,  au  point  ou  il  est  impossible  de  le  distinguer  de  celui  des  specialistes. 
L’appareil  HOT  foumit  aux  FC  une  propriete  intellectuelle  de  la  Couronne  destinee  a  former  du 
personnel  a  reagir  en  situation  de  menace,  alors  qu’aucun  mode  de  formation  dans  ce  domaine 
n’existait  auparavant. 

Plans  futurs  : 

Plusieurs  ameliorations  peuvent  etre  apportees  a  l’appareil  HOT  afm  d’accroitre  Tutilite  de  la 
version  actuelle.  L’imagerie  visuelle  peut  etre  rendue,  de  fa9on  optimale,  en  fonction  de  la  surface 
de  projection  existante.  De  plus,  le  temps  que  met  un  stagiaire  a  reagir  a  une  menace  peut  etre 
sounds  a  Tinstructeur  de  fa9on  plus  Liable.  Enfin,  Tapparence  des  balles  tra9antes  et  des  trainees 
de  missiles  peut  etre  plus  realiste  et  plus  diversifiee. 

Les  futures  generations  de  Tappareil  HOT  offrant  des  conditions  de  vision  nocturne  et  une 
simulation  de  vision  nocturne  a  l’aide  de  lunettes  (ou  une  stimulation  de  vision  nocturne)  doivent 
faire  l’objet  d’une  etude  de  validation. 
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1  Introduction 


The  Canadian  Forces  (CF)  operates  aircraft  in  theatres  where  they  are  threatened  by  missiles  and 
gunfire.  Skilled  operation  of  the  aircraft  is  a  key  element  in  mitigating  these  threats.  In  particular, 
timely,  coherent  and  accurate  reporting  of  the  threats  is  vital  to  defeating  them. 
Accordingly,  the  Hercules  Observer  Trainer  (HOT)  was  developed  to  support  the  training  of 
surface-to-air  threat  reaction  and  reporting.  This  document  reports  an  evaluation  of  the  training 
effectiveness  of  the  HOT. 

1.1  Project  background 

Aircrew  operating  the  CC-130  transport  aircraft  in  combat  areas  visually  scan  their  environment 
to  detect  and  coordinate  reaction  to  threats.  The  threats  include  small  arms,  anti-aircraft  artillery 
(AAA),  surface  to  air  missiles  (SAMs),  helicopters  (HELO),  and  fixed  wing  threats.  Observers  in 
the  aircraft  report  the  existence  of  the  threats,  provide  advice  on  the  required  reaction,  and  report 
the  location  of  the  threat.  Their  reports  must  be  fast,  accurate,  and  consistent  for  the  crew  and 
aircraft  to  respond  in  a  timely  and  appropriate  manner  [1]. 

Currently,  the  CF  has  limited  means  to  train  aircrew  in  the  recognition  and  avoidance  of  these 
threats.  Computer  simulation  of  ground  threats  is  not  currently  available,  live  training  is  very 
primitive,  and  the  measures  of  effectiveness  are  limited  to  instructor  observations.  The 
commander  of  8  Wing  therefore  raised  a  Statement  of  Operational  Capability  Deficiency 
(SOCD),  endorsed  by  1  Can  Air  Div,  to  obtain  training  devices  to  address  the  deficiency  [2],  In 
parallel  with  this  SOCD,  the  Canadian  Forces  Aerospace  Warfare  Centre  (CFAWC)  tasked 
DRDC  Toronto  with  developing  a  prototype  device  to  provide  CC-130  loadmasters  with  threat 
reaction  training  [3]. 

In  response  to  the  CFAWC  tasking,  DRDC  conducted  an  analysis  of  the  training  system 
deficiencies  facing  CC-130  operators.  The  analysis  confirmed  the  high  priority  of  the  threat 
reaction  training  [4].  Following  from  that,  DRDC  performed  a  task  analysis  and  training 
requirements  analysis  to  set  the  requirements.  DRDC  specified  a  technical  solution  that  was 
implemented  under  a  series  of  contracts  using  CFAWC  funds  [5  -  8], 

The  tasking  required  that  the  ensuing  prototype  be  evaluated  for  its  training  effectiveness.  The 
experiment  reported  in  this  document  provides  an  evaluation  of  the  training  effectiveness 
of  the  HOT. 


1.2  HOT  description 

Analysis  of  the  training  requirement  and  meetings  with  CC-130  operators  led  to  the  design  of  the 
training  device.  The  HOT  consists  of  three  components:  the  trainee  station;  the  instructor  / 
operator  station  (IOS);  and  the  synthetic  environment. 

The  trainee  station  presents  a  visual  display  of  computer-generated  imagery  depicting  the  view 
from  the  CC-130  paratrooper  door.  The  display  enables  the  trainee  to  view  any  objects  that  are 
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visible  from  the  windows  of  the  aircraft.  To  enable  the  reporting  of  the  threats  and  dialogue  with 
the  instructor  /  operator,  the  student  station  has  an  intercom  for  voice  communication.  Figure  1 
shows  the  trainee  station  with  a  simple  paratrooper  door  and  window  mock-up  that  serves  to 
orient  and  constrain  the  observer’s  view. 


Figure  1  The  HOT  Trainee  Station 


The  IOS  provides  the  functions  to  start  the  trainer,  load  and  edit  scenarios,  control  execution  of 
the  simulation,  and  monitor  performance.  The  instructor  /  operator  has  voice  communications 
with  the  trainee  and  can  control  the  threat  reaction  of  the  trainee’s  aircraft.  The  IOS  also  includes 
a  briefing  and  debriefing  capability.  Using  this  capability,  the  instructor  /  operator  can  provide 
briefings  on  the  training  scenario  and  the  anticipated  threats  as  well  as  debriefing 
performance  by  replaying  the  scenario,  and  providing  feedback  in  terms  of  effectiveness  and 
performance.  Figure  2  shows  the  IOS  as  it  appears  during  a  simulation  run. 
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Figure  2  The  HOT  Instructor  Operator  Station 


The  synthetic  environment  has  two  chief  functions.  The  first  is  to  create  the  flight  dynamics  of  the 
trainee’s  aircraft,  thus  providing  the  trainee  station  with  the  visual  experience  of  inhabiting  the 
aircraft  reacting  to  the  threats.  The  HOT  provides  for  two  sources  of  flight  dynamics.  One  source 
is  a  computer-generated  forces  (CGF)  software  package  integral  to  the  HOT  that  simulates  the 
flight  of  the  CC-130.  When  the  HOT  is  operating  in  stand-alone  mode,  the  trainee’s  viewpoint  is 
anchored  to  the  side  of  the  CC-130  simulated  by  the  CGF  system.  Through  the  use  of  distributed 
simulation  technologies,  the  flight  dynamics  controlling  the  trainee  station  viewpoint  can  also 
come  from  sources  external  to  the  HOT.  When  operating  in  this  networked  mode,  the  trainee’s 
viewpoint  can  be  controlled  by  another  CGF,  a  flight  simulator,  or  even  a  real  aircraft.  The 
synthetic  environment  is  also  responsible  for  presenting  the  threats.  The  dynamics  of  the  threats 
and  the  events  of  the  scenario  are  controlled  by  the  synthetic  environment.  The  threats  can  be 
generated  by  the  CGF  integral  to  the  HOT,  or  by  other  external  sources. 
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2  Experiment 


As  called  for  in  the  tasking,  an  experiment  was  conducted  to  evaluate  the  HOT.  The  goal  of  the 
experiment  was  to  determine  if  the  HOT  is  an  effective  device  for  training  threat  reaction. 
Specifically,  answers  to  three  questions  were  sought. 

Is  the  HOT  a  valid  simulation  of  the  threat  reaction  environment  in  the  CC-130? 

A  valid  simulation  of  the  threats,  the  observer’s  situation,  and  the  aircraft  movements  is  sought  so 
that  the  experience  provided  by  the  HOT  is  representative  of  the  actual  threat  reaction  situation.  It 
is  important  to  note  that  the  validity  of  the  simulation  components  is  relevant  to  the  extent  that 
they  affect  the  perceptions  and  actions  required  to  properly  perform  the  threat  reaction.  For 
example,  the  trainee’s  viewpoint  should  be  within  the  target  aircraft  so  that  the  dynamics  and 
perspective  of  the  threats  in  consistent  with  that  found  in  the  real  world.  On  the  other  hand,  the 
colour  of  SAM  rocket  motor  smoke,  for  example,  does  not  have  bearing  on  the  reaction  (vice 
recognition)  to  the  threats. 

To  answer  this  validity  question,  an  experimental  technique  called  backward  transfer  was 
employed  [9]  in  which  the  performance  of  experienced  CC-130  loadmasters,  referred  to  as 
experts  in  this  report,  were  compared  to  new  loadmasters,  referred  to  as  novices.  If  the  simulation 
is  a  valid  depiction  of  the  threat  reaction  environment,  expertise  in  threat  reaction  should  be 
evident  in  performance.  The  experts  should  initially  outperform  the  novices. 

Does  the  HOT  support  learning? 

A  training  device  must  foster  development  of  its  targeted  skill.  To  answer  this  question,  the 
changing  performance  of  the  novices  will  be  examined.  If  the  novice’s  performance  is  observed 
to  improve  with  use  of  the  HOT,  this  will  be  evidence  that  learning  is  occurring. 

The  performance  of  experts  will  also  be  monitored  for  improvements  but  conclusions  may  be 
harder  to  draw.  A  lack  of  improvement  by  the  experts  could  be  attributable  to  a  ceiling  effect.  In 
this  case,  a  lack  of  improvement  would  occur  because  they  already  know  all  that  the  HOT  is 
capable  of  training.  On  the  other  hand,  if  expert  performance  is  seen  to  improve,  this  could  arise 
from  improved  proficiency  at  making  threat  calls  or  to  learning  of  discrepancies  between 
performing  the  task  in  the  simulator  and  in  the  aircraft. 

Does  learning  in  the  HOT  transfer  positively  to  the  operational  environment? 

The  purpose  training  devices  is  to  improve  performance  in  the  operational  environment.  If 
training  with  the  device  does  not  transfer,  or  leads  to  worse  performance  (negative  transfer),  then 
the  device  should  not  be  used  for  training.  From  the  standpoint  of  experimental  logic,  the  most 
straightforward  and  compelling  way  to  answer  this  question  would  be  to  compare  the 
performance  of  trained  and  untrained  crew  members  in  a  live-flying  environment  exposed  to  real 
threats.  Obviously,  there  are  many  reasons  why  this  cannot  be  done.  Instead,  the  validity  of  the 
device  will  be  used  to  argue  that  the  skills  should  transfer  to  the  operational  environment.  The 
backward  transfer  of  training  logic  will  be  used  to  argue  that  if  the  device  allows  experts  to  show 
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their  expertise  in  the  simulator,  then  one  should  expect  that  novices  who  acquire  skills  in  the 
simulator  should  be  able  to  demonstrate  them  in  the  operational  environment  [9]. 


2.1  Methodology 

An  experiment  to  answer  the  above  questions  was  conducted  in  February  and  March  2008,  at 
CFB  Trenton.  An  exercise  run  director  from  DRDC  Toronto  oversaw  the  running  of  the  trial.  A 
contracted  technician,  trained  on  the  operation  of  the  FIOT,  controlled  the  simulation.  An 
instructor  from  426  Squadron  provided  instruction  to  the  experimental  subjects. 

2.1.1  Subjects 

Sixteen  members  of  the  CF  served  as  subjects  in  the  experiment.  Eight  of  the  subjects  were 
experienced  CC-130  aircrew,  qualified  for  the  tactical  airlift  mission  and  familiar  with  making 
threat  calls.  In  addition,  most  of  them  were  also  instructors  at  the  Operational  Training  Unit,  426 
Squadron.  These  subjects  were  defined  as  the  experts.  Eight  additional  CF  members  were  drawn 
from  the  population  of  aircrew  attending  the  CC-130  Basic  Loadmaster  Course  at  CFB  Trenton. 
These  subjects  were  not  yet  qualified  for  the  CC-130  tactical  airlift  mission  and  they  had  not 
completed  the  threat  call  training  included  in  the  course  they  were  attending.  These  subjects  were 
defined  as  the  novices. 

The  subjects  were  tasked  by  426  Squadron  to  participate.  However,  their  data  were  used  only  if 
they  provided  informed  consent  for  its  use.  In  addition,  all  subjects  were  informed  of  their  ability 
to  withdraw  from  participation  in  the  trial  without  consequences.  All  subjects  provided  their 
consent  and  none  withdrew  from  the  trial. 

2.1.2  The  task 

Threat  call  standard  operating  procedures  are  published  in  the  Standard  Manoeuvre  Manual  [1]. 
This  publication  identifies  the  terminology  and  syntax  for  a  variety  of  threats.  Although  not 
explicitly  addressed  in  the  published  standard  operating  procedures,  consideration  of  the  tactical 
situation  and  consultation  with  subject  matter  experts  reveals  that  the  calls  must  be  prompt  and 
accurate. 

This  experiment  considered  only  SAM  and  AAA  threats.  Tables  1  and  2  present  the  threat  call 
criteria  for  SAM  and  AAA  threats  in  this  experiment.  Each  call  provides  four  (in  the  case  of 
SAMs)  or  five  (in  the  case  of  AAA)  types  of  information,  followed  by  updates  as  the  tactical 
situation  allows.  The  threat  is  reported  either  as  a  SAM  or  AAA  with  no  addition  detail  on  type. 
The  threat  direction  is  given  relative  to  the  aircraft  as  facing  toward  the  nose  of  the  aircraft.  The 
clock  position  is  a  horizontal  frame  of  reference,  with  12  o’clock  being  at  the  nose  of  the  aircraft. 
Distance  is  given  in  nautical  miles  from  the  aircraft.  An  example  of  a  SAM  call  would 
be  “SAM  right  4  o’clock  3  miles  -  overhead”.  An  example  of  an  AAA  call  would  be  “JINK  AAA 
left  9  o’clock  1  mile  -  still  firing  -  still  firing-  cease  firing”. 
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Type  of  Information 

Possible  Values 

Threat 

SAM 

Direction 

LEFT  or  RIGHT 

Clock  Position 

1  -  12  O’CLOCK 

Distance 

x  MILES 

Update  clock  position  and  distance 

1-12  O’CLOCK  and  x  MILES  or 
OVERHEAD 

Table  1  SAM  Threat  Call 


Type  of  Information 

Possible  Values 

Action 

JINK 

Threat 

AAA 

Direction 

LEFT  or  RIGHT 

Clock  Position 

1-12  O’CLOCK 

Distance 

x  MILES 

Update 

STILL  FIRING  or  CEASE  FIRING 

Table  2  AAA  Threat  Call 


2.1.3  Apparatus 

Vega  Prime  ™  visual  simulation  software  with  the  FX  ™  special  effects  module  rendered  the 
imagery  at  60  Flz  for  the  trainee  station.  The  geometric  field  of  view  generated  by  the  software 
provided  a  scene  180°  wide  by  135°  high.  The  imagery  was  displayed  using  an  OmniFocus  ™ 
FIAL  SX-3  projector  with  a  1400  x  1050  resolution  and  an  Immersive  Display  Solutions 
Immersive  Dome  ™  200C  2.2  meter  inflatable  hemispherical  projection  screen.  The  subjects 
viewed  the  screen  through  a  hole  cut  in  a  plywood  screen  that  constrained  their  view  to  that 
available  through  the  CC-130  paratrooper  door  normally  used  by  loadmasters  watching  for  threats 
The  screen  shown  in  Figure  1  is  a  subsequent  metal  version.  An  example  of  the  imagery 
presented  to  the  subjects  is  presented  in  Figure  3.  The  image  in  Figure  3  has  a  “fish-eye” 
appearance  because  it  is  intended  to  be  projected  on  a  hemispherical  screen.  The  scene  appears 
undistorted  when  projected  on  the  hemispherical  screen  and  observed  from  the  design  eyepoint. 
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Figure  3  Example  of  imagery  at  the  student  station  before  projection  onto  the  dome. 


To  use  the  intercom  to  make  their  threat  reports,  the  subjects  wore  a  headset  with  a  boom 
microphone.  To  report,  subjects  depressed  a  push-to-talk  switch.  They  could  hear  instructions  and 
corrective  feedback  through  the  headset. 

2.1.4  Experimental  design 

The  experiment  was  a  mixed  design  where  each  subject  attended  two  experimental  sessions  that 
were  separated  by  one  to  two  weeks.  At  the  start  of  the  first  session,  the  run  director  explained  the 
purpose  of  the  experiment,  informed  the  subject  that  he  could  withdraw  from  the  experiment  at 
any  time  without  repercussions,  and  obtained  informed  consent  for  the  use  of  the  subject’s  data. 
The  HOT  system  operator  and  the  instructor  then  provided  a  briefing  on  the  HOT,  reviewed  the 
correct  threat  calls,  and  demonstrated  two  examples  each  of  SAM,  AAA,  and  helicopter  (HELO) 
threats  using  the  HOT.  The  structure  of  the  blocks  of  trials  was  then  explained  to  the  subject  and 
any  questions  raised  were  answered.  The  subject  then  completed  six  blocks  of  trials.  The  structure 
of  the  trials  is  presented  in  Table  3.  The  first  two  blocks  consisted  entirely  of  SAMs  to  facilitate 
learning  of  the  calls.  The  AAA  threats,  which  require  a  slightly  more  complicated  call,  were 
introduced  in  Blocks  3  and  4.  The  final  two  blocks  consisted  of  a  mix  of  SAM,  AAA,  and  HELO 
threats.  The  subjects  were  not  trained  on  reporting  the  HELO  threats  and  were  therefore 
instructed  that  no  call  was  required. 

The  position  of  the  threats  relative  to  the  aircraft  was  varied  to  both  ensure  that  the  subjects  had  to 
make  every  possible  call  and  to  encourage  them  to  scan  the  full  extent  of  their  assigned  arc.  The 
threats  appeared  at  three  different  clock  positions  and  three  different  ranges.  The  combination  of 
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three  ranges  at  three  locations  resulted  in  9  threats  in  each  block.  In  the  blocks  5  and  6,  the  mixed 
blocks,  the  HELO  threats  appeared  in  two  of  the  nine  threat  locations.  Because  threat  calls  were 
not  required  for  the  HELO  threats,  two  of  the  possible  locations  were  not  the  subject  of  threat 
calls  in  these  blocks. 

Due  to  the  complexity  of  defining  each  simulation  scenario,  each  subject  saw  the  same  block.  The 
threats  within  a  block  occurred  one  at  a  time  and  were  spaced  between  10  and  60  seconds  apart. 
Completion  of  a  block  required  approximately  7  minutes  and  the  overall  session  ran 
under  90  minutes. 


Table  3  Blocks  of  trials 


Block 

Threat 

Details 

1 

SAM 

9  SAM 

2 

SAM 

9  SAM 

3 

AAA 

9  AAA 

4 

AAA 

9  AAA 

5 

MIX 

4  SAM,  3  AAA,  2  HELO 

6 

MIX 

4  SAM,  3  AAA,  2  HELO 

The  initial  block  of  each  set  of  threats  (i.e.  blocks  1,  3,  and  5)  were  unscored  tutored  blocks.  In 
these  blocks  the  instructor  would  halt  the  simulation  and  coach  the  subject  on  the  threat  call,  as 
required.  In  the  other  blocks,  the  subject  completed  the  block  without  instructor  intervention. 

One  to  two  weeks  after  the  first  session,  the  time  varying  according  to  the  availability  of  the 
subject,  the  subjects  completed  a  second  experimental  session.  The  second  session  omitted  the 
introduction  and  collection  of  consent  but  was  otherwise  identical  to  the  first  session. 

The  accuracy  and  syntax  of  the  subject’s  threat  reports  were  graded  by  the  run  director.  The 
programmed  nature  of  the  threats  allowed  the  run  director  to  know  the  correct  reports  in  advance. 
It  was  therefore  possible  to  compare  the  subject’s  report  to  the  geometrically  and  syntactically 
correct  report  required  during  run  time.  The  subject’s  report  and  score  for  each  trial  were 
recorded  on  paper. 

The  accuracy  of  the  call  was  scored  against  the  known  true  values.  A  trial  was  scored  as  accurate 
if  all  values  were  correct  ±  one  value.  For  example,  if  a  threat  was  located  at  3  o’clock,  the  call 
would  be  scored  as  accurate  if  the  subject  reported  2,  3,  or  4  o’clock.  Other  values  would  be 
scored  as  incorrect. 

The  syntax  of  the  call  was  scored  as  correct  if  the  threat  information  was  provided  in  the  correct 
order  and  used  the  correct  terminology,  as  described  in  Table  1.  Additional  information  before  the 
call,  omission  of  information,  information  out  of  order,  or  incorrect  terminology  resulted  in  the 
syntax  for  the  trial  being  scored  as  incorrect.  Updates  to  the  calls  were  not  scored. 
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2.2  Data  analysis 

The  collected  data  were  analyzed  to  answer  questions  of  the  validity  of  the  HOT,  the  achievement 
of  learning  in  the  HOT,  and  whether  there  is  evidence  of  training  transfer.  All  collected  data  were 
used  in  the  analyses.  One  expert  was  unable  to  complete  the  second  session  and  those  data  were 
therefore  not  available  to  include  in  the  analyses. 

First,  a  multivariate  analysis  of  variance  (MANOVA)  was  conducted  to  determine  if  any 
differences  existed  amongst  the  experimental  conditions.  Using  a  significance  level  of  a  =  .05,  a 
three-way  MANOVA  (expertise  x  session  x  threat)  indicated  that  there  were  significant  effects  of 
the  experimental  conditions.  Wilk’s  Lambda  and  associated  F  values  for  the  effects  of  expertise 
(F  3i34=  3.82,  p  =  .018),  session  (F  3,34=  5.30,  p  =  .004),  and  threat  type  (F33 4=  6.55,  p  =  .001), 
justified  closer  examination  of  the  data  using  univariate  methods. 

A  repeated  measures  analysis  of  variance  (ANOVA)  performed  on  the  syntax  data 
detected  effects  significant  at  the  .05  level.  Specifically,  syntax  improved  in  the  second 
session  (F  1;  61  =  12.27,  p  =  .004).  The  same  analysis  performed  on  the  accuracy  data  also 
detected  significant  effects.  Accuracy  improved  in  the  second  session,  (F  ,  61  =  22.05,  p  <  .001), 
and  AAA  threats  were  reported  more  accurately  than  SAM  threats  (F  i,6i=  8.59,  p=  .011). 
The  interaction  of  expertise  and  session  also  produced  a  significant  effect  on 
accuracy,  (F  ^  6i  =  5.00,  p  =  .043). 


Given  the  significant  effects  in  the  syntax  and  accuracy  data,  these  data  were  then  examined  using 
planned  comparisons.  The  research  questions  regarding  validity  and  learning  could  be  answered 
with  a  comparison  to  the  novice  performance  in  the  first  session.  To  answer  these  questions,  the 
first  session  novice  performance  was  therefore  treated  as  the  control  condition  in  a  Dunnetf  s 
multiple  comparisons  with  a  control  condition  [10].  In  each  case  it  was  predicted  that 
performance  would  be  better  than  performance  in  the  first  session  of  novices,  so  the  results  were 
evaluated  against  a  one-sided  criterion  of  a  <  .05.  The  result  in  these  comparisons  was  that  the 
other  conditions  were  significantly  better  than  the  first-session  novice  performance.  These  results 
are  discussed  in  more  detail  below. 

2.2.1  Evidence  of  validity 

The  data  were  analyzed  for  evidence  that  the  HOT  is  a  valid  simulation  of  calling  threats  in  the 
CC-130.  Specifically,  the  performance  of  experts  and  novices  were  compared  in  the  first  session 
of  training.  If  performing  threat  calls  in  the  HOT  is  not  representative  of  making  threat  calls  in  a 
tactical  setting,  the  performance  of  the  experts  and  novices  should  not  differ.  However,  if  it  is 
representative,  then  one  should  expect  experts  to  outperform  the  novices.  The  syntax  and 
accuracy  data  collected  in  the  first  training  session,  collapsed  over  threat  type  and  analyzed  with 
the  Dunnetf  s  multiple  comparison  with  the  control,  both  showed  that  the  experts  indeed 
significantly  out-performed  novices  in  the  first  session  at  the  a  <  .05  level.  The  difference 
between  novices  and  experts  in  the  first  session  is  readily  visible  in  Figure  4  and  Figure  5. 
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2.2.2  Evidence  of  learning 


Next,  the  data  were  examined  to  confirm  that  learning  was  occurring.  The  specific  comparison 
tested  whether  the  novices  improved  over  the  course  of  the  two  sessions.  The  analyses  were 
planned  to  conserve  statistical  power  by  only  considering  the  data  from  the  novices  and  forgoing 
examination  of  the  expert  data  because  of  the  ambiguities  associated  with  the  possible  outcomes, 
as  described  in  section  2.0. 

The  syntax  and  accuracy  data,  collapsed  over  threat  type  and  analyzed  with  the  Dunnett’s 
multiple  comparison  with  the  control,  both  showed  that  the  novices  improved  in  session  2  relative 
to  session  1  at  the  a  <  .05  level.  Again,  the  difference  between  novice  performance  in  sessions  1 
and  2  is  visible  in  Figure  4  and  Figure  5. 

2.2.3  Evidence  of  transfer 

The  best  measure  of  transfer  of  training  would  be  improved  performance  in  the  operational 
environment.  Collecting  that  sort  of  data  was  not  an  option  for  this  study.  Therefore,  the 
performance  of  the  novices  was  examined  to  determine  whether  training  in  the  FIOT  would  result 
in  their  performance  resembling  that  of  experts.  The  comparison  reported  in  the  Evidence  of 
Validity  sections  found  that  experts  were  significantly  better  than  novices  initially.  During 
training,  the  experts  improved,  as  did  the  novices.  At  the  end  of  training,  the  novices  were  making 
syntactically  correct  and  accurate  threat  calls  as  frequently  as  the  experts  (the  superior  accuracy 
of  the  novices  is  not  statistically  reliable  at  the  a  <  .05  level).  In  Figure  4  and  5,  one  can  see  the 
novices  overcome  an  initial  deficit  in  the  first  session,  finishing  the  training  session  with 
performance  equal  to  that  of  the  experts. 

2.3  Discussion 

The  experiment  was  conducted  to  determine  the  validity  of  the  FIOT,  whether  subjects  learned 
from  the  FIOT,  and  whether  the  learning  in  the  FIOT  was  transferable.  The  experiment  provided 
two  90-minute  training  sessions  to  novices  and  experts.  Each  training  session  provided  structured 
training  and  feedback  to  the  subjects,  allowing  them  to  learn  and  practice  making  threat  calls 
against  26  SAM  and  24  AAA  threats. 

The  syntax  and  accuracy  data  generated  statistically  significant  and  consistent  results.  The  data 
suggest  that  the  FIOT  is  tapping  the  skills  involved  in  making  threat  calls.  This  is  demonstrated  by 
the  reliably  better  syntax  and  accuracy  exhibited  by  the  experts  at  the  start  of  training.  If  the  HOT 
did  not  depend  on  the  skills  involved  in  making  threat  calls,  one  would  expect  no  difference 
between  experts  and  novices.  It  is  also  apparent  that  learning  is  possible  through  the  use  of  the 
HOT.  The  syntax  and  accuracy  of  the  novices’  threat  calls  were  observed  to  improve  through 
training.  Finally,  training  with  the  HOT  enables  the  novices  to  deliver  expert  performance. 
Following  the  two  sessions  of  training,  their  syntax  and  accuracy  was  indistinguishable  from  that 
of  the  experts. 
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2.3.1  Limitations  of  the  trial 


The  results  of  the  trial  are  reliable,  but  the  trial  was  subject  to  limitations  caused  by  unexpected 
circumstances,  practical  considerations,  and  trade-offs  made  in  the  experimental  design.  These 
limitations  temper  the  conclusions  drawn  and  suggest  where  additional  HOT  development  or 
instructional  methods  could  increase  the  benefits  possible  with  the  HOT. 

The  threats  did  not  accurately  depict  any  particular  threat  weapon  system.  The  use  of  real  threats 
against  real  aircraft  was  not  practical  and  the  use  of  classified  data  was  judged  by  the  author  to  be 
unnecessary  to  determine  the  training  value  of  the  HOT.  The  modelling  of  the  surface  to  air 
threats  was  medium  fidelity,  with  only  one  SAM  and  one  AAA  threat  presented  in  different 
locations.  This  reduced  variability  may  have  lead  to  somewhat  faster  learning  than  might  be 
found  with  greater  variety  of  threats.  Any  overestimation  of  the  speed  of  learning  is  likely  to  be 
small,  however,  given  the  high  degree  of  commonality  in  the  appearance  of  AAA  and  SAM 
threats.  On  a  related  note,  the  training  of  threat  reaction  should  not  be  contused  with  threat 
recognition.  The  HOT  was  not  configured  or  tested  to  train  the  recognition  of  specific  threat 
types,  but  rather  the  recognition  of  the  existence  of  a  threat  class  and  the  appropriate  response. 

The  claim  that  learning  in  the  HOT  transfers  to  the  operational  environment  depends  on  argument 
rather  than  data  collected  from  live  flight  exposed  to  real  threats.  The  argument  goes  as  follows: 
Given  that  the  simulation  has  been  shown  to  be  a  valid  representation  of  the  operational 
environment,  data  showing  that  training  leads  to  novices  and  experts  performing  equivalently  in 
the  simulation  is  synonymous  with  showing  that  the  training  leads  to  novices  and  experts 
performing  equivalently  in  the  operational  environment.  This  experimental  tactic  of  analogical 
transfer  is  perhaps  necessary  in  high-risk  situations  [11],  but  its  necessity  does  not  make  it 
compelling.  Nevertheless,  the  live  data  that  would  be  ideal  from  a  scientific  standpoint  (live 
threats  in  a  live  flying)  are  not  available  due  to  cost  and  safety  concerns.  Additional  support  for 
transfer  might  be  obtained  if  the  HOT  is  employed  by  aircrew  who  subsequently  experience  these 
threats. 

The  inability  of  the  tested  version  of  the  HOT  to  reliably  capture  the  time  to  respond  to  the  threats 
compromises  estimates  of  the  device’s  training  benefit.  There  may  be  a  speed  -  accuracy  tradeoff. 
Specifically,  the  subjects  might  be  responding  more  slowly  to  improve  the  syntax  and  accuracy  of 
their  responses  [12,  13].  It  did  not  appear  to  be  the  case  to  the  instructors,  operators,  or 
experimenters;  rather  the  opposite  was  their  belief.  Nevertheless,  without  solid  data,  this  criticism 
cannot  be  positively  refuted.  It  can  be  pointed  out,  however,  that  even  such  a  result  would 
demonstrate  learning  and  sensitivity  to  conditions  and  their  performance.  Furthermore,  it  would 
still  count  as  improvement  in  accordance  with  the  current  training  manuals  since  they  mandate 
only  correct  syntax  and  not  a  performance  standard. 

The  training  sessions  were  structured  to  gather  evidence  bearing  on  the  experimental  questions. 
Structuring  the  training  sessions  differently  might  lead  to  better  training.  For  example,  subjects 
were  trained  one  at  a  time.  In  an  operational  training  setting,  students  typically  benefit  from 
watching  one  another  perform,  and  learning  from  other’s  successes  and  failures.  Also,  throughout 
the  training  sessions,  the  subjects  were  denied  any  coaching  or  feedback  during  the  second  block 
of  any  threat  type.  In  an  operational  training  setting,  subjects  who  failed  to  master  the  threat  call 
in  the  first  block  would  be  given  additional  coaching. 
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Also,  not  all  threats  were  presented  in  the  trial.  Rotary  and  fixed  wing  threat  calls  were  not 
evaluated.  Lasers  are  also  not  represented.  The  HOT  hardware  and  software  has  the 
potential  to  present  these  threats  and  can  evaluate  responses  to  them.  However,  until  such 
functionality  is  fully  implemented  and  training  scenarios  developed,  the  HOT  is  unable  to  address 
all  threat  call  training. 

Finally,  this  trial  did  not  attempt  to  represent  all  threat  situations  and  environments.  All 
engagements  were  in  daylight  over  a  desert  terrain.  The  subjects  were  not  trained  how  to  perform 
the  task  when  wearing  night  vision  goggles  or  during  degraded  atmospheric  conditions.  It  should 
not  be  assumed  that  the  HOT  can  train  the  performance  of  the  task  under  these  because  these 
conditions  can  change  the  appearance  of  the  threats.  Misrepresentations  of  the  threats  could  lead 
to  no  training  or  even  negative  training.  Modifications  to  the  HOT  might  be  required  to  deliver 
this  training. 

The  HOT’s  visual  system  is  not  able  to  provide  all  the  visual  cues  present  in  the  real  world.  It 
cannot  provide  the  same  resolution,  brightness  variability,  and  terrain  detail  that  are  seen  in  the 
real  world,  for  example.  Thus,  although  threat  reaction  can  be  trained  using  the  HOT,  threat 
detection  and  the  time  to  respond  in  the  HOT  are  not  necessarily  representative  of  what  will  be 
obtained  in  the  real  world. 


2.4  Conclusions 

The  HOT  was  developed  to  provide  simulation-based  threat  reaction  training  to  CC-130 
loadmasters  and  others  who  work  in  the  CC-130  cargo  compartment.  The  HOT  utilizes  a 
simulated  crew  station  to  provide  students  with  a  simulated  view  from  a  CC-130  being  engaged 
by  AAA  and  SAMs.  Structured  training  materials  and  scenarios  enable  instructors  to  teach  threat 
calls  to  the  students  in  a  dynamic,  interactive,  and  controlled  simulation.  The  experiment  revealed 
that  the  HOT  is  a  valid  representation  of  the  threat  environment,  as  shown  by  expert  aircrew 
initially  out-performing  novices  in  the  threat  reaction  task.  The  HOT  demonstrated  its  ability  to 
train  and  to  lead  to  expert  performance  when  the  performance  of  novices  improved  and  came  to 
be  indistinguishable  from  that  of  the  experts. 
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3  Recommendations 


•  The  HOT  has  demonstrated  the  ability  to  improve  the  threat  call  performance  of  novices  to 
that  of  experts  in  the  course  of  two  90-  minute  sessions.  The  HOT  should  therefore  be 
considered  for  use  in  the  training  of  CC-130  loadmasters.  The  timing  and  structure  of  the 
training  sessions  used  in  this  trial  should  be  revised  by  the  CC-130  training  community, 
taking  into  consideration  the  current  threat  scenarios,  prevailing  administrative  constraints, 
and  local  knowledge.  It  should  be  anticipated  that  different  training  regimens  will  be 
required  for  users  with  different  levels  of  experience,  such  as  novices  versus  pre-deployment 
refresher  training  for  operational  crews. 

•  The  CC-130  community  should  consider  establishing  performance  standards  for  threat  calls. 
Standards  for  timeliness,  accuracy  of  the  calls,  and  the  syntactical  correctness  should  be 
considered.  The  availability  of  such  standards  will  provide  guidance  to  the  development  of 
training  scenarios  and  the  amount  of  HOT  training  required. 

•  Several  improvements  should  be  made  to  the  HOT  to  increase  the  utility  of  the  existing 
version  of  the  device.  The  visual  imagery  should  be  optimally  rendered  for  the  existing 
projection  surface.  Also,  the  time  for  a  student  to  respond  to  a  threat  should  be  made 
available  to  the  instructor  in  a  reliable  manner.  Finally,  the  appearance  of  the  tracer  fire  and 
missile  trails  should  be  made  more  realistic  and  variable. 

•  If  the  HOT  is  taken  into  use  for  operational  training,  opportunities  for  confirming  the 
positive  transfer  of  learning  from  training  to  operations  should  be  sought.  Although  it  is  not 
expected  that  enough  data  could  be  collected  for  a  statistical  analysis  to  have  sufficient 
power  to  show  a  quantitative  training  effect,  interviews  could  be  recorded  that  would  at  least 
document  striking  qualitative  effects. 

•  Any  future  generations  of  the  HOT  that  provide  night  viewing  conditions  and  night 
vision  goggle  simulation  (or  night  vision  goggle  stimulation)  should  be  subject  to  a 
validation  study. 
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[Enter  list  here,  if  applicable.  If  not,  delete  the  page.] 


AAA 

Anti  -  Aircraft  Artillery 

ANOVA 

Analysis  of  Variance 

CF 

Canadian  Forces 

CFAWC 

Canadian  Forces  Aerospace  Warfare  Centre 

CGF 

Computer  Generated  Forces 

DRDC 

Defence  Research  &  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 
Management 

HELO 

Helicopter 

HOT 

Hercules  Observer  Trainer 

IOS 

Instructor  Operator  Station 

MANOVA 

Multivariate  Analysis  of  Variance 

R&D 

Research  &  Development 

SAM 

Surface  to  Air  Missile 

SOCD 

Statement  of  Operational  Capability  Deficiency 
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